Path probing using an edge completion ratio

ABSTRACT

Various embodiments pertain to communication network systems. In particular, various embodiments relate to multi-path probing in communication network systems that can be used to estimate the complete topology of the network. A method includes receiving data at a source node from a tracerouting probe in the network. The tracerouting probe detects edges. The method also includes calculating an edge completion ratio based on the edges. In addition, the method includes terminating the probe when the edge completion ratio is greater than or equal to a threshold.

BACKGROUND Field

Various embodiments pertain to communication network systems. Inparticular, various embodiments relate to multi-path probing incommunication network systems that can be used to estimate the completetopology of the network.

Description of the Related Art

A variety of networks use multi-path routing to connect source nodes andtarget nodes within the network. Multi-path routing is a routingtechnique of using multiple alternative paths through a network betweentwo points in the network. The path created in the network includes atleast one intermediate network node, located between the source node andthe target node, and an edge, which is used to connect the variousnetwork nodes in the path. In multi-path routing, each path includes atleast one edge and at least one network node. The multiple alternativepaths in the network can often at least partially overlap, so that thepaths share at least one edge and/or at least one node.

Some networks using multi-path routing can employ tracerouting.Tracerouting is a network diagnostic tool that may be used to recordroutes through the network. Tracerouting involves the sending ofinformation from each intermediate network node in the network back tothe source node. This signaling can allow the source node to track arouting of a signal through the network to the target node.

Using multi-path routing can yield a variety of benefits to the network,such as fault tolerance, increased bandwidth, or improved security.Building a complete network topology, however, may require a completetracing or mapping of all possible routes in the network. Such acomplete tracing or mapping may utilize a significant amount of networkresources, including time and bandwidth.

SUMMARY

A method, in certain embodiments, may include receiving data at a sourcenode from a tracerouting probe in a network. The tracerouting probe candetect edges. The method may also include calculating an edge completionratio based on the edges. In addition, the method may includeterminating the probe when the edge completion ratio is greater than orequal to a threshold.

According to certain embodiments, an apparatus may include at least onememory including computer program code, and at least one processor. Theat least one memory and the computer program code may be configured,with the at least one processor, to cause the apparatus at least toreceive data at a source node from a tracerouting probe in a network.The tracerouting probe can detect edges. The at least one memory and thecomputer program code may also be configured, with the at least oneprocessor, at least to calculate an edge completion ratio based on theedges. In addition, the at least one memory and the computer programcode may also be configured, with the at least one processor, at leastto terminate the probe when the edge completion ratio is greater than orequal to a threshold.

An apparatus, in certain embodiments, may include means for receivingdata at a source node from a tracerouting probe in a network. Thetracerouting probe may detect edges. The apparatus may also includemeans for calculating an edge completion ratio based on the edges. Inaddition, the apparatus may include means for terminating the probe whenthe edge completion ratio is greater than or equal to a threshold.

According to certain embodiments, a non-transitory computer-readablemedium encoding instructions that, when executed in hardware, perform aprocess. The process may include receiving data at a source node from atracerouting probe in a network. The tracerouting probe may detectedges. The process may also include calculating an edge completion ratiobased on the edges. In addition, the process may include terminating theprobe when the edge completion ratio is greater than or equal to athreshold.

According to certain embodiments, a computer program product encodinginstructions for performing a process according to a method includingreceiving data at a source node from a tracerouting probe in a network.The tracerouting probe may detect edges. The method may also includecalculating an edge completion ratio based on the edges. In addition,the method includes terminating the probe when the edge completion ratiois greater than or equal to a threshold.

A method, in certain embodiments, may include receiving identificationinformation of a first network node in a multi-path route. The methodcan also include receiving identification information of a secondnetwork node in the multi-path route. In addition, the method includesdetermining an identification of an edge based on the identificationinformation of the first network node and the identification informationof the second network node.

According to certain embodiments, an apparatus may include at least onememory including computer program code, and at least one processor. Theat least one memory and the computer program code may be configured,with the at least one processor, to cause the apparatus at least toreceive identification information of a first network node in amulti-path route. The at least one memory and the computer program codemay also be configured, with the at least one processor, to cause theapparatus at least to receive identification information of a secondnetwork node in the multi-path route. In addition, the at least onememory and the computer program code may be configured, with the atleast one processor, to cause the apparatus at least to determine anidentification of an edge based on the identification information of thefirst network node and the identification information of the secondnetwork node.

An apparatus, in certain embodiments, may include means for receivingidentification information of a first network node in a multi-pathroute. The apparatus may also include means for receiving identificationinformation of a second network node in the multi-path route. Inaddition, the apparatus may include determining an identification of anedge based on the identification information of the first network nodeand the identification information of the second network node.

According to certain embodiments, a non-transitory computer-readablemedium encoding instructions that, when executed in hardware, perform aprocess. The process may include receiving identification information ofa first network node in a multi-path route. The process may also includereceiving identification information of a second network node in themulti-path route. In addition, the process may include determining anidentification of an edge based on the identification information of thefirst network node and the identification information of the secondnetwork node.

According to certain embodiments, a computer program product encodinginstructions for performing a process according to a method includingreceiving identification information of a first network node in amulti-path route. The method may also include receiving identificationinformation of a second network node in the multi-path route. Inaddition, the method may include determining an identification of anedge based on the identification information of the first network nodeand the identification information of the second network node.

BRIEF DESCRIPTION OF THE DRAWINGS

For proper understanding of the invention, reference should be made tothe accompanying drawings, wherein:

FIG. 1 illustrates a communication system according to certainembodiments.

FIG. 2 illustrates a communication system according to certainembodiments.

FIG. 3 illustrates a flow diagram according to certain embodiments.

FIG. 4 illustrates equations according to certain embodiments.

FIG. 5 illustrates a flow diagram according to certain embodiments.

FIG. 6 illustrates a communication system according to certainembodiments.

FIG. 7 illustrates a communication system according to certainembodiments.

FIG. 8 illustrates a flow diagram according to certain embodiments.

FIG. 9 illustrates a system according to certain embodiments.

DETAILED DESCRIPTION

Certain embodiments can estimate a complete network topology. A completenetwork topology may be defined as having traced or mapped all of theunique edges in the network. A topology completion ratio may be used,which can indicate a percentage of the complete network topology thatmay be mapped and reported to a source node. A tracerouting probe may beused to facilitate estimating of the complete network topology. Theprobe may in some embodiment continuously or discretely map, trace, orcapture paths through the network.

In certain embodiments an estimated complete network topology may beused to monitor an end-to-end network performance. In a multi-pathnetwork environment, it may be helpful from a monitoring and diagnosticperspective to cover a majority of network paths through whichend-to-end traffic flows. This can improve the diagnostic results of thenetwork, and allow the evaluator of the results, for example, a networkoperator, to make decisions based on the results.

The multi-path probing may be optimized in such a way that the probe canbe stopped when a given topology completion target is reached. Forexample, once a completion threshold of the network topology is met, theprobing may be suspended or stopped. Having to trace a complete networktopology may be cumbersome, and use a significant amount of networkresources. As such, an efficient network topology estimation process maysave the network both time and resources. In certain embodiments, thenetwork topology tracing may include a self-evaluation method toestimate the network topology, without needing any operatorinterference.

In certain other embodiments, multi-path probing may be used to findpaths from itself to any given endpoint. A tracerouting probe can beused to collect network performance metrics along the paths to the givenendpoint, and trigger the reporting of performance metrics from thedifferent intermediate nodes on the path. Such performance metrics, suchas packet loss and latency, may then be used to detect transit trafficproblems at least one edge or at least one node on the probe paths. Thetracerouting probe in some embodiments can provide for fast pathprobing.

Once tracerouting is complete, or the tracerouting probe has beenterminated, a proper visualization of the network may viewed by a userto investigate any problems associated with the network, for example.While tracerouting can be focused on different paths between nodes, incertain embodiments it may be helpful to determine the identification ofthe nodes on the different paths. Further, it may also be helpful todetermine the relation of the different nodes on the path to the devicesin the network. A device may be, for example, a user equipment, a mobileterminal, a base station, an access point, a router, and/or a server,

In certain embodiments, just because two different packets may havedifferent paths that include different nodes, having different IPaddresses, may not mean that packets have to go through differentdevices. Each device, for example a router or a server, may havemultiple IP addresses assigned to different interfaces. In other words,each device may have more than one node with its own unique IP address.Deciding which interface to use may, in some embodiments, be based on arouting protocol and/or other additional logic, such as load balancing.

FIG. 1 illustrates a communication system diagram according to certainembodiments. Specifically, FIG. 1 illustrates a network using multi-pathrouting between a source node 110 and a target node 160. Source node 110and target node 160, for example, may be network nodes, access nodes,eNodeBs, servers, hosts, or any of the other access or network nodesdiscussed herein. In certain embodiments, source node 110 may use atracerouting probe. The tracerouting probe may be used to detectdifferent routing paths between source node 110 and target node 160. Inone example, the probe may begin at source node 110, and then move tonetwork node 120 through edge 111.

An edge may be a wireless, wired, or cloud based connection between twonodes that may be traced or mapped by the probe. When all of the uniqueedges in a network are found, the network topology may be consideredcomplete. Once the probe reaches network node 120, it can trigger thesending of a message from network node 120 to source node 110. Themessage may include performance metrics or other information related toedge 111, such as timing information and/or packet loss information. Thetracerouting probe may also trigger sending source node 110 any othertype of information related to any of the edges or network nodes in thenetwork.

After network node 120, the tracerouting probe may then map or trace oneof three different paths. It may continue to network node 130, usingedge 121, network node 140, using edge 123, or network node 150, usingedge 122. Assuming that the probe moves to network node 130, using edge121, the tracerouting probe may then again have several different pathsit can trace or map. For example, from network node 130, thetracerouting probe may trace edge 131 and continue to target node 160.Alternatively, the tracerouting probe may continue to network node 140via edge 133, and then to target node 160 via edge 141.

As discussed above, certain embodiments may include the sending oftracerouting probing packets. The nodes along a given traces path, forexample intermediate and target nodes, can respond to the probingpackets by sending information about the node to source node 110.Similarly, once a probe packet reached target node 160, target node 160will report information related to target node 160 to source node 110.

Once a tracerouting probe reaches target node 160, the traceroutingprobe may start routing another path from source node 110 to target node160. In certain embodiments the tracerouting probe may continuouslytrace or map network paths until the estimated network topology reachesa certain threshold. While some of the mapped or traced routes mayoverlap previously mapped or traced paths, some paths may include new orunique edges. New or unique edges may include edges that have not beenpreviously traced by the tracerouting probes. For example, traceroutingprobe may move from source node 111 to network node 120, and then tonetwork node 150, through edge 122, and to target node 160 through edge151. Edges 122 and 151 are unique edges that have not previously beentraced or mapped by the probe.

A complete network topology may include tracerouting of all networknodes 110, 120, 130, 140, and 150. Every node can have its own uniqueidentifier, for example, IP address, domain name, and location orposition information on the path that can be used to identify thenetwork node. A complete network topology may also include traceroutingall unique edges in the network. FIG. 1 only illustrates a partiallycompleted topology of the network, in which edges 111, 121, 122, 123,131, 133, 141, 142, and 151 have been traced or mapped by thetracerouting probe.

Tracing a complete network topology may require a significant amount oftime, and demand a significant amount of network resources. FIG. 2illustrates a communication system according to certain embodiments. Inparticular, FIG. 2 illustrates a multi-path network using a traceroutingprobe. The probe attempts to trace paths from source nodes 210, 211, totarget nodes 220, 221. FIG. 2 illustrates 3312 cached unique paths thathave been traced or mapped by the tracerouting probe, and reported tosource nodes 210, 211. As previously discussed, having to map a completenetwork topology may can be cumbersome on the network, and demand alarge amount of network resources.

Certain embodiments may therefore include an estimation of the completenetwork topology. The complete network topology may be estimated using anetwork topology completion ratio. The network topology completion ratiomay be estimated using an edge completion ratio. Performance metricsreported by the tracerouting probe may be converted to, or put in theform of, a per edge metric. For example, the message from thetracerouting probe to the source node may include edge latency and/oredge packet loss. Increasing the number of mapped unique edges can allowfor an improved detection rate of traffic problems along the pathsbetween the source node and the target node.

The edge completion ratio may be an estimated ratio of the number ofunique edges mapped and reported to the source node. When the edgecompletion ratio is a hundred percent, the source node contains acomplete mapping of the unique edges in the network, and the networktopology completion ratio can also be said to be at a hundred percent.In certain embodiments, the edge completion ratio may be defined as thenumber of unique edges found in the network divided by the estimatedtotal number of unique edges.

In certain embodiments, the edge can be identified based on the networknodes to which the edge is connected. In other words, the identificationof the entering node of the edge and the identification of the exitingnode of the edge are used to identify the edge itself. In someembodiments, node information may not be available, for example, thenode is timed out or the node has a private IP address. As describedbelow, in such embodiments the node information may be determined basedon the prefix of the node located before the unavailable node and afterthe unavailable node. Similarly, the prefix of the entering node and theprefix of the exiting node may be combined to determine the identity ofthe edge. A source node may determine whether the edge is unique basedon an edge identification. In addition, the edge identification can berecorded at the source node using information related to the enteringnode and the exiting node of a given edge. The edge identification canbe used to mark the edges.

When the complete network topology is not known by the source node, oneway to calculate the edge completion ratio may be based onmark-release-recapture (MRR) methods. MRR methods, for example, mayinclude capturing, marking, and releasing a first portion of a givenpopulation. Then, a second portion of the population is captured, andthe number of marked members of the second portion of the population canbe counted. Since the number of marked members of the population in thesecond portion should be proportional to the number of members in theentire population, an estimate of the total number of members within apopulation size may be obtained. MRR methods may therefore be used toestimate both an entire population, as well as the number of a givenmember within the population.

FIG. 3 illustrates a flow diagram according to certain embodiments. Instep 310, a signal may be transmitted between a source node and a targetnode via multi-path routing. Examples of the multi-path routing can beseen in FIGS. 1 and 2. A tracerouting probe may then be used to trace ormap at least one edge and at least one node on the path between thesource node and the target node. Upon reaching a network node, atracerouting probe may trigger the transmission of information or datafrom the network node to the source node, which details at least oneedge performance metric. For example, the at least one edge metric mayrelate to edge latency and/or edge packet loss.

In step 320, the source node receives the data from the traceroutingprobe in a network. As previously described, the tracerouting probe mapsor traces at least one edge and at least one network node in thenetwork. For every x number of captured paths, x being configurable bythe network operator, the source node may identify all unique edgeswithin the x number of paths, and compare them with the previouslyrecorded edges. The number of recaptured edges may also be taken intoaccount by the source node. If the same edge was recaptured multipletimes, the recaptured edge will only be counted as one edge. The numberof newly identified or unique edges will be added to the recorded edgesfor the next calculation. The source node may store the recorded edges,for example, in a list.

An estimated edge completion ratio can then be calculated based on theedges, as shown in step 330, using a MRR method. For example, Schnabeland Schumacher-Eschmeyer estimation method may be used as the preferredMRR methods. The estimation equation may in certain embodiments be asfollows:

$N^{\prime} = \frac{\sum\limits_{i - 2}^{c}\; \left( {n_{i}\left( {\sum\limits_{j = 1}^{i - 1}\; n_{j}} \right)}^{2} \right)}{\sum\limits_{i = 2}^{c}\; \left( {m_{i}\left( {\sum\limits_{j = 1}^{i - 1}\; n_{j}} \right)} \right)}$

N′ may be an estimation of total number of unique edges from the sourcenode to the target node. n_(i) may equal the total number of uniqueedges caught at test i, while m_(i) may equal the number of recapturedunique edges during test i. c may equal the total number ofcalculations.

Based on the above calculation, the edge completion ratio may becalculated by dividing the number of unique edges found in the networkby the estimated total number of unique edges calculation, an estimationof the total number of unique edges may be determined. This edgecompletion ratio may then be used to estimate a topology completionratio, as shown in step 340. The edge completion ratio and/or thetopology completion ratio can then be compared to a threshold set by anetwork operator. For example, the threshold may be 95% in certainembodiments. As shown in step 350, if the edge completion ratio and/orthe topology completion ratio is greater than or equal to a threshold,then the tracerouting probe may be terminated.

The threshold may be predetermined by the operator. The higher thethreshold, the more network resources may be used for the mapping or thetracing of the complete network topology. If the threshold is too low,on the other hand, then the source node may not be able to accuratelygage the complete topology of the network.

In certain embodiments, the confidence limits of Schnabel andSchumacher-Eschmeyer estimates can be calculated using the reciprocal of[N′[1/(1/N′±t(0.05, c−1) S.E.(1/N′))], or by using the formula(N′±t(0.05, c−1) S.E.(N′)). The latter formula (N′±t(0.05, c−1)S.E.(N′)) may provide for a closer value, including a higher minimum orlower positive maximum, than the former formula. FIG. 4 illustratesequations according to certain embodiments. Specifically, FIG. 4illustrates S.E. (N′) 410 and S.E. (1/N′) 420 used in calculating theconfidence limits of Schnabel and Schumacher-Eschmeyer.

In some embodiment, the edge completion ratio may be filtered so as toremove hops with only one single edge. Edges that are always included byall paths may be ignored. For example, a first edge, from the sourcenode to its default gateway, may be ignored if the source node has onlyone default gateway. Ignoring certain edged may be done in order toavoid any distortions to the estimation, and to increase the accuracy ofthe estimation.

In addition, in certain embodiments a sliding window approach may beused in which only the most recent data can be used for the estimation.The sliding window approach may be used when the source node isconstantly probing paths, without having to complete a self-containedtest. The sliding window approach includes that the completion ratioestimation can be based on paths collected from the recent time window,known as the sliding window, at any time. This may allow for calculatingthe ratio more frequently, as well as allowing for the adjustment of thewindow side.

As discussed above, multi-path traceroute algorithm produces a set of atleast partially differing paths from a probing source to a target. Oncethe tracerouting is completed, a graph of all the different individualpaths, which shows the pairing of the nodes in multiple paths that mayrepresent the same network IP interfaces, may be produced. At thisphase, paths are only sequences of nodes, whose identification may noteven be known. While the IP addresses of the Nodes may be known, noother additional information about the nodes may be known. For example,at least of the Node IDs, Node IP addresses, Edge IDs, domain namesystem (DNS), performance metrics, or public identification records, maynot be known.

In certain embodiments, a node identifier may be calculated for allnodes in the determined tracerouting paths, even if a graph representingthe traceroute may be incomplete. In other words, node identificationmay occur even when some nodes on the path are still missing. In someembodiments, an IP address of a node may be public, such as the IPaddresses of nodes routed in the global Internet. When the node IPaddress is public, the node identity may simply be the node IP address.

In some other embodiments, however, the IP address of a node may beprivate in accordance with, for example, Request for Comment 1918 (RFC1918), for Internet Protocol version 4 (IPv4). The IP address may, inanother embodiment, be local according to RFC 4193 for Internet Protocolversion 6 (IPv6). If the IP address of a node is private, it may behelpful to attach a network scope to the IP address, since such an IPaddress may be assigned in multiple distinct internal networks. Todefine the network scope, an identifier of the closest preceding nodewith a public IP address is used. If, however, no such preceding nodewith a public IP address exists, the identity of the probing source, forexample a poller, agent, or other source, is used. In such anembodiment, the path node examined may be part of the probe's internalnetwork, and can be mapped to internal monitoring systems.

In yet another embodiment, the IP address of a node may not bediscovered or may be unresponsive. There may, however, be reason tobelieve that a node exists, for example, because there are signs of atraffic hop at that location, despite the lack or responsiveness of thenode. When the node has no IP address or is not responsive, theidentifier of the node may include the following: the identity or prefixof the closest preceding responsive node in the path, the identity orprefix of the closest subsequent responsive node in the path, and aposition in the sequence of unresponsive nodes between the preceding andsubsequent responsive nodes.

In some embodiments, the target node, may in certain embodiments, not beresponsive. This can leave the routing path open, without a definitivetermination node. In such embodiments, when the target is notresponsive, the identifier of the node may include the identity orprefix of the closets preceding responsive node in the path, and/or theidentity or prefix of the closest subsequent responsive node in thepath. Once all of the nodes in the path are identified, the individualpaths may be aligned to nodes with same identifier. These nodes maybecome junction points in a resulting path graph.

FIG. 5 illustrates a flow diagram according to certain embodiments.Specifically, FIG. 5 illustrates a flow diagram for determining theidentification of an edge based on the determined or calculatedidentification and/or prefix of the entering or exiting nodes. FIG. 5,therefore, may illustrate additional features that may be incorporatedinto FIG. 3 and/or FIG. 8. In step 510, the source node may receiveidentification information of a first network node in a multi-pathroute. The source node may receive the information from a traceroutingprobe that can be used for mapping or tracing the network topology. Instep 520, the source node may receive identification information of asecond network node in the multi-path route. Based on the identificationinformation of the first network node and the second network node, thesource node can determine an identification of the edge, as shown instep 530.

The edge identification determined in step 530 may be stored or recordedin the source node. The source node may use the edge identification todetermine whether an edge is unique, to estimate the edge completionratio of the network. As shown in FIG. 3, the edge completion ratio canbe used to estimate the complete network topology.

In certain embodiments, in which a Node's IP address may be private, thediscovered nodes in the multi-path traceroute may be mapped to datastored in a monitoring system. The stored data may be part of aninternal network, which may require a username and/or password toaccess. The stored data may cover at least part of the internal networkof a single device, and may be stored by the user or by automateddiscovery, which may itself use node reduplication logic. In otherwords, the data may be used to determine to which device a discoverednode belongs. The stored data can include an array or a list of IPaddresses per device being monitored. Therefore, the data may be used tomap discovered IP addresses to existing devices in a network monitoringsystem.

All of the IP addresses in the stored data and/or the discovered nodesmay be unique in some embodiments. In such embodiments, a node ID ismatched to a particular IP address in the stored data, which includes adevice's IP array or list. In other embodiments, however, the storeddata and/or the discovered nodes may include at least one duplicate IPaddress. For example, several discovered nodes may be matched to asingle device. This may be because the internal network may have severalsimilar network with the same private IP addresses or subnets. In otherwords, it may not be possible to use the stored data of the managementsystem to map nodes to concrete devices because multiple devices mayshare the same IP address.

In the embodiment in which multiple devices may share the same IPaddress, each device in the management system may access interfaceinformation, which includes information about the neighbors of thedevice. The interface information for the at least one neighbor devicemay be stored along with other stored data of the management system. Insome embodiments, the IP address or subnet information of at least oneneighboring device may be matched to the IP address of at least onenode, or those traceroute nodes nearest to the at least one node.Devices which have multiple interfaces, for example, multiple relatedsubnets, which correspond with the IP addresses of neighbors, can beused for mapping.

As discussed above, certain embodiments may be provided with devices, oran aggregation of nodes within such devices, which have the same countof interfaces as previous traceroute node count. In other words, theparticular node ID may be matched to the IP address of a given device,and devices and node IDs are matched. Those interfaces may then bemapped to the correct links, edges, nodes, and/or devices.

Each device or node may have two different interfaces, ingress oregress. An ingress interface may be an interface for data that isentering the device or node, and can be represented on the right side ofa link or an edge connected to such a device or node. An egressinterface, on the other hand, may be an interface for data that isexiting a device or a node. For example, if a first device and a seconddevice are connected via an edge, the right side of the edge canrepresent the location of the ingress interface of the second device,while the left side of the edge may represent the egress of the firstdevice.

The Node ID which represents the traceroute node may be the IP addressof the ingress interface of the node. While probing is able to identifyingress interfaces of a node or a device, egress interfaces may in someembodiments not be detected by the probe. A single device or node, forexample, has multiple egress interfaces, and it may be helpful todetermine which of the nodes is connected via a link or an edge to theingress interface. Routing and topology information included in thestored data of the management system may be used to determine themapping of the egress interface.

If the identification of the egress interface cannot be determined basedon the routing and topology information included in the managementsystem, the identification of the egress interface may be determinedbased on the identity most similar to the ingress identity. Thesimilarity of the addresses may be determined based on the similarity ofthe IP addresses of two different interfaces. For example, if the IPaddress of the ingress interface is 10.0.0.1, and the only two other IPaddresses that are known are 10.0.0.5 and 10.0.0.25, the more similar IPaddress may be 10.0.0.5. This IP address may then be assigned to theegress interface. In some other embodiments, determining the similaritybetween the ingress and egress interface may include the use of a subnetmask.

In certain embodiments, getting a packet from the probe to the internetmay include a first node and a second node. In such an embodiment, afirst interface eth0 may be used to access the second node, rather thana second interface eth1, which may be used to access the internet fromthe second node. The right side of the edge or link between the firstand second node may be recognized as an ingress interface of the secondnode. The IP address of the second node may be, for example, IP10.0.1.2. In other embodiments the IP address of the second node mayhave any other Node ID or Node IP address. The ingress interfaceidentification may have been determined or discovered by the probeduring tracerouting.

The left side of the edge and/or the link may then be identified. Theend point on the left side of the edge may be identified based on thenext hop address, for example, the identification of the second node,and/or all of the enumerated interfaces of the current node, such as thefirst node. The subnet for a particular interface may then be resolved,based on at least the IP address and the subnet mask. A subnet mask maybe a 32-bit number that masks an IP address, and divides the IP addressinto network address and host address. The ID of the first node and IDof the second node may then be fitted to the resolved subnet address.However, if the IDs of the first and the second nodes do not fit, theabove may be repeated with the next available interface in the device.If the identifications of the first and the second nodes fit, theinterface may be chosen as the egress interface for Node 1.

In certain embodiments, the goal of the above matching between the NodesIDs and the subnet address may be to determine the end points of a givenedge or link. In other words, one the ingress interface and the egressinterface of the nodes may be determined, the Node IDs of all of thenodes on the probing path may be determined. In addition, the ingressand egress interface can also be used to determine the number of nodesin a given device.

Once the traceroute leaves the internal network, in some embodiments,the device information contained in a network management system may nolonger be available to a lack of administrative access. In suchembodiments, a user may still want to determine the identification ofthe nodes, edges, devices, and the number of nodes located in thedifferent devices. In some embodiments, a node IP address a node DNSname may be used as the Node ID, and/or connectivity between at leasttwo different nodes may be used to determine the Node ID. In certainembodiments, as discussed above, traceroute probing discover the IPaddress of the ingress interface in which the probing traffic enters thedevice, but may not determine the IP address of the egress interface.There may therefore not be any information available for which theegress interface connects to the next hop device.

In certain embodiment in which the device has multiple ingressinterfaces, the traceroute graph may present each ingress interface IPas a separate node. In some embodiments, this may make the graph morecomplex than it should be and not representative of the network. Todetermine which traceroute nodes can be aggregated, several features maybe included. For example, the path probing packets may not enter thesame device more than once, which may limit the search scope ofcandidate nodes for aggregation. In some embodiments, devices may havemultiple virtual routers and packets. In such embodiments, each virtualrouter may be treated as a separate device node.

Yet in other embodiments, each ingress interface may only be connectedto one egress interface. This can limit the search scope of candidatenodes for aggregation. In other embodiments, however, two or moreinterfaces may share the same subnet. For example, nodes to beaggregated may be internet service provider (ISP) devices or cloudservice provider devices. In some embodiments, systematic namingconventions may be commonly used for router interfaces in backbonenetworks.

In certain embodiments, therefore, traceroute nodes may qualify foraggregation when the traceroute nodes do not belong to the same path,have a common succeeding node, and/or share the same DNS name suffix,and their name prefixes may fall into the same naming convention. Theabove embodiments allow the Node ID, and the number of Nodes within agiven device, to be determined based on connectivity between nodes.

FIG. 6 illustrates a communication system according to certainembodiments. In particular, FIG. 6 illustrates a graph of a multi-pathnetwork using a tracerouting probe. The probe attempts to trace pathsfrom source node 610 to target node 620. The embodiment of FIG. 6 is agraph of paths before node aggregation has occurred. Node aggregationmay mean aggregating the nodes in the graph into their respectivedevices. In other words, the nodes in the graphs have not yet beenrelated to their corresponding devices, meaning that only nodes, notcorresponding devices, are shown in the embodiment of FIG. 6.

FIG. 7 illustrates a communication system according to certainembodiments. In particular, FIG. 7 illustrates a graph of a multi-pathnetwork using a tracerouting probe. The probe attempts to trace pathsfrom source node 710 to target node 720. Unlike the embodiment of FIG.6, the embodiment of FIG. 7 is a graph of the multi-path network afternode aggregation. In other words, the nodes in FIG. 7 have been relatedto their corresponding devices, and the corresponding devices, ratherthan individual nodes, are illustrated in the graph. As can be seen inFIG. 7, several devices include a plurality of network nodes.

FIG. 8 illustrates a flow diagram according to certain embodiments. Step810 may describes receiving data at a source node from a traceroutingprobe in a network. The data may include information about at least onenetwork node. The information about the at least one network node maythen be used to determine the identification of the at least one node,as shown in step 820. As shown in FIG. 5, the identification of the atleast one node may be used to determine the identification of the edge.In step 830, the identification of the at least one network node may beused to determine an identification of at least one device. There may bea plurality of the at least one network node in one device. In otherwords, a single device may include multiple network nodes.

In certain embodiments, the identification of the at least one node maybe based on an identification of a first network node, which precedesthe at least one network node, an identification of a second networknode, which is subsequent to the at least one network node, and/or aposition of the network node between the first and second network nodes.In other embodiments, the identification of the at least one networknode or the at least one device may be determined using informationderived from a network management system, which may be an internalnetwork. Further, in some embodiment, the identification of the at leastone network node may be determined based on the ingress and egressinterfaces.

FIG. 9 illustrates a system according to certain embodiments. It shouldbe understood that each block in FIGS. 1-8 may be implemented by variousmeans or their combinations, such as hardware, software, firmware, oneor more processors and/or circuitry. In one embodiment, a system mayinclude several devices, such as, for example, a network node 910. Thesystem may include more than one network node 910. The network node maybe a source node, a target node, an intermediate node, an access node, abase station, an eNodeB, 5G NB, server, host, router, or any of theother access or network node discussed herein.

Network node 910 may include at least one processor or control unit ormodule, indicated as 911. At least one memory may be provided asindicated by 912. The memory may include computer program instructionsor computer code contained therein. One or more transceivers 913 may beprovided, and each network node may also include an antenna,respectively illustrated in 914. Although only one antenna each isshown, many antennas and multiple antenna elements may be provided toeach of the devices. Other configurations of the network node may beprovided. For example, network node 910 may be additionally configuredfor wired communication, in addition to wireless communication, and insuch a case antennas 914 may illustrate any form of communicationhardware, without being limited to merely an antenna.

Transceivers 913 may independently, be a transmitter, a receiver, orboth a transmitter and a receiver, or a unit or device that may beconfigured both for transmission and reception. The transmitter and/orreceiver (as far as radio parts are concerned) may also be implementedas a remote radio head which is not located in the device itself, but ina mast, for example. The operations and functionalities may be performedin different entities, such as nodes, hosts or servers, in a flexiblemanner. In other words, division of labor may vary case by case. Onepossible use is to make a network node deliver local content. One ormore functionalities may also be implemented as virtual application(s)in software that can run on a server.

In some embodiments, an apparatus, such as network node 910, may includemeans for carrying out embodiments described above in relation to FIGS.1-8. In certain embodiments, at least one memory including computerprogram code can be configured to, with the at least one processor,cause the apparatus at least to perform any of the processes describedherein.

According to certain embodiments, an apparatus 910 may include at leastone memory 912 including computer program code, and at least oneprocessor 911. The at least one memory 912 and the computer program codeare configured, with the at least one processor 911, to cause theapparatus 910 at least to receive data at a source node from atracerouting probe in a network. The tracerouting probe detects edges.The at least one memory 912 and the computer program code may also beconfigured, with the at least one processor 911, to also cause theapparatus 910 at least to calculate an edge completion ratio based onthe edges. In addition, the at least one memory 912 and the computerprogram code may be configured, with the at least one processor 911, tocause the apparatus 910 at least to terminate the probe when the edgecompletion ratio is greater than or equal to a threshold.

According to certain embodiments, an apparatus 910 may include at leastone memory 912 including computer program code, and at least oneprocessor 911. The at least one memory 912 and the computer program codeare configured, with the at least one processor 911, to cause theapparatus 910 at least to receive identification information of a firstnetwork node in a multi-path route. The at least one memory 912 and thecomputer program code may also be configured, with the at least oneprocessor 911, to also cause the apparatus 910 at least to receiveidentification information of a second network node in the multi-pathroute. In addition, the at least one memory 912 and the computer programcode may be configured, with the at least one processor 911, to causethe apparatus 910 at least to determine an identification of an edgebased on the identification information of the first network node andthe identification information of the second network node.

Processor 911 may be embodied by any computational or data processingdevice, such as a central processing unit (CPU), digital signalprocessor (DSP), application specific integrated circuit (ASIC),programmable logic devices (PLDs), field programmable gate arrays(FPGAs), digitally enhanced circuits, or comparable device or acombination thereof. The processors may be implemented as a singlecontroller, or a plurality of controllers or processors.

For firmware or software, the implementation may include modules or unitof at least one chip set (for example, procedures, functions, and soon). Memory 912 may independently be any suitable storage device, suchas a non-transitory computer-readable medium. A hard disk drive (HDD),random access memory (RAM), flash memory, or other suitable memory maybe used. The memories may be combined on a single integrated circuit asthe processor, or may be separate therefrom. Furthermore, the computerprogram instructions may be stored in the memory and which may beprocessed by the processors can be any suitable form of computer programcode, for example, a compiled or interpreted computer program written inany suitable programming language. The memory or data storage entity istypically internal but may also be external or a combination thereof,such as in the case when additional memory capacity is obtained from aservice provider. The memory may be fixed or removable.

The memory and the computer program instructions may be configured, withthe processor for the particular device, to cause a hardware apparatussuch as network node 910, to perform any of the processes describedabove (see, for example, FIGS. 1-8). Therefore, in certain embodiments,a non-transitory computer-readable medium may be encoded with computerinstructions or one or more computer program (such as added or updatedsoftware routine, applet or macro) that, when executed in hardware, mayperform a process such as one of the processes described herein.Computer programs may be coded by a programming language, which may be ahigh-level programming language, such as objective-C, C, C++, C#, Java,etc., or a low-level programming language, such as a machine language,or assembler. Alternatively, certain embodiments may be performedentirely in hardware.

As discussed above, certain embodiments can utilize an edge completionratio to estimate a complete network topology. Once the edge completionratio is determined to be equal to or greater than a certain threshold,the tracerouting probing of the network may be terminated. This can beused to preserve network resources, and make the tracing or mapping ofthe network topology more efficient.

Estimating the complete network topology can help monitor end-to-endnetwork performance. Monitoring, evaluation, and diagnosing theend-to-end network performance can help a network operator or a certainnetwork node make various decisions relating to the performance of thenetwork. For example, the node may make latency decisions and shiftnetwork resources based on the estimated network topology. Thus, theabove embodiments provide significant improvements to the functioning ofa network and/or to the functioning of the nodes or computers within thenetwork.

In addition, certain embodiment may allow for determining not only thenodes in a network, but also the devices associated with such nodes. Inparticular, it may be possible to determine the number of nodes withineach device, and thereby, the number of devices in the network. This mayallow for a graph produced by a tracerouting probe to be reduced to agraphs that includes Nodes IDs, Edge IDs, and different devices.

The features, structures, or characteristics of certain embodimentsdescribed throughout this specification may be combined in any suitablemanner in one or more embodiments. For example, the usage of the phrases“certain embodiments,” “some embodiments,” “other embodiments,” or othersimilar language, throughout this specification refers to the fact thata particular feature, structure, or characteristic described inconnection with the embodiment may be included in at least oneembodiment of the present invention. Thus, appearance of the phrases “incertain embodiments,” “in some embodiments,” “in other embodiments,” orother similar language, throughout this specification does notnecessarily refer to the same group of embodiments, and the describedfeatures, structures, or characteristics may be combined in any suitablemanner in one or more embodiments.

One having ordinary skill in the art will readily understand that theinvention as discussed above may be practiced with steps in a differentorder, and/or with hardware elements in configurations which aredifferent than those which are disclosed. Therefore, although theinvention has been described based upon these preferred embodiments, itwould be apparent to those of skill in the art that certainmodifications, variations, and alternative constructions would beapparent, while remaining within the spirit and scope of the invention.The communication systems discussed above may be implemented in ThirdGeneration Partnership (3GPP) technology, such as Long Term Evolution(LTE), LTE-advanced (LTE-A), 3^(rd) generation technology (3G), 4^(th)generation technology (4G), 5^(th) generation technology (5G), or anyother IP Multimedia System (IMS).

Partial Glossary

-   S.E. standard error-   N′ an estimation for total number of unique edges from the source    node to the target node.-   n₁ the total number of unique edges caught at test i.-   m_(i) the number of recaptured unique edges during test i.-   c the total number of calculations.

We claim:
 1. A method comprising: receiving data at a source node from atracerouting probe in a network, wherein the tracerouting probe detectsedges; calculating an edge completion ratio based on the edges; andterminating the tracerouting probe when the edge completion ratio isgreater than or equal to a threshold.
 2. The method according to claim1, further comprising: receiving identification information of a firstnetwork node in a multi-path route; receiving identification informationof a second network node in the multi-path route; and determining anidentification of the edge based on the identification information ofthe first network node and the identification information of the secondnetwork node.
 3. The method according to claim 1, further comprising:estimating a topology completion ratio based on the edge completionratio.
 4. The method according to claim 3, wherein the edge completionratio is calculated by a number of captured unique edges divided by anestimated total number of the unique edges.
 5. The method according toclaim 3, wherein the edge completion ratio is computed using amark-release-recapture equation.
 6. The method according to claim 5,wherein the mark-release-recapture equation comprises at least one of aSchnabel equation or a Schumacher-Eschmeyer method.
 7. The methodaccording to claim 1, wherein the tracerouting probe continuouslycaptures the edges from the source node to a target node.
 8. The methodaccording to claim 1, wherein the threshold equals ninety five percent.9. The method according to claim 1, wherein the threshold ispredetermined by an operator.
 10. The method according to claim 1,further comprising: transmitting a signal between the source node and atarget node via multi-path routing.
 11. The method according to claim10, wherein the multi-path routing comprises a self-evaluation process,and wherein the self-evaluation process comprises the traceroutingprobe.
 12. The method according to claim 1, wherein the traceroutingprobe determines data metrics based on the edges.
 13. The methodaccording to claim 1, further comprising: storing the data received fromthe tracerouting probe at the source node.
 14. An apparatus, comprising:at least one memory comprising computer program code; at least oneprocessor; wherein the at least one memory and the computer program codeare configured, with the at least one processor, to cause the apparatusat least to: receive data at a source node from a tracerouting probe inthe network, wherein the tracerouting probe detects edges; estimate atopology completion ratio based on the edges; and terminate the probewhen the topology completion ratio is greater than or equal to athreshold.
 15. The apparatus according to claim 14, wherein the at leastone memory and the computer program code are also configured, with theat least one processor, to cause the apparatus at least to: receiveidentification information of a first network node in a multi-pathroute; receive identification information of a second network node inthe multi-path route; and determine an identification of the edge basedon the identification information of the first network node and theidentification information of the second network node.
 16. The apparatusaccording to claim 14, wherein the estimating based on the edgescomprises an edge completion ratio.
 17. The apparatus according to claim14, wherein the edge completion ratio is calculated by a number ofcaptured unique edges divided by an estimated total number of the uniqueedges.
 18. The apparatus according to claim 14, wherein the edgecompletion ratio is computed using a mark-release-recapture equation.19. The apparatus according to claim 14, wherein the at least one memoryand the computer program code are also configured, with the at least oneprocessor, to cause the apparatus at least to: sending the data betweenthe source node and a target node via multi-path routing.
 20. Theapparatus according to claim 19, wherein the multi-path routingcomprises a self-evaluation process, and wherein the self-evaluationprocess comprises a tracerouting probe.
 21. A non-transitorycomputer-readable medium encoding instructions that, when executed inhardware, perform a process, the process comprising: receiving data at asource node from a tracerouting probe in the network, wherein thetracerouting probe detects edges; calculating an edge completion ratiobased on the edges; and terminating the tracerouting probe when the edgecompletion ratio is greater than or equal to a threshold.